Ekaitz's tech blog:
I make stuff at ElenQ Technology and I talk about it

As the project’s goal was to create a Gopher client, it was time to understand something about the protocol and read the RFC. No need for you to know the protocol to understand what I’m going to say here. I think I already did the difficult part for you.

Understand some Gopher

Gopher is a really simple protocol (this doesn’t mean I implemented it correctly anyway). It’s assumed to work on top of TCP (default port is 70) and it’s as simple as creating a socket, sending the selector string to it followed by a line terminator¹, and reading everything from it until it closes. That’s in most of the cases how it works.

It has two major ways to access the data:

1. Text mode, which is used in most of the queries, needs the client to read from the socket until a line with a single dot (.) appears. Then the connection is closed.

2. Binary mode, expects the client to read from the socket until the server closes it.

Easy-peasy.

Gopher is a stateless protocol and that helps a lot during the creation of the client. There’s no need to retain data or anything related.

Selector strings are what client wants to see. In order to know what selections are possible, Gopher defines an specific text format that works as a menu, and it’s called, unsurprisingly, Menu.

Menus have a description of the content, the address or hostname, the port, the selector string, and a number that indicates the type of each of its elements separated by a TAB (aka \t character). Each element in one line¹.

Pay attention to the fact that each menu entry contains an address and a port, that means it can be pointing to a different server!

The type further than making the client choose between binary and text mode also gives the client information about what kind of response it’s going to get from it: if it’s a menu, an image, an audio file… It also says if the element is a search endpoint².

Yes, Gopher supports searches!

Well, Gopher supports tons of things because the only rule is that all the logic is on the server side. You can do whatever you want, if you do it on the server.

Searching is as simple as asking for a text document, but it adds also the search query to the equation. During a search, the client needs to send the selector string to select the endpoint and then the search string (separated by a TAB character).

There are some points more but this is more than enough for the moment.

Let’s make something work.

Make Gopher queries

Before jumping to Clojure, lets make sure that we understood how this works with some simple text queries. In a UNIX-like terminal you can do the following to navigate the Gopherverse:

exec 5<>/dev/tcp/tilde.team/70
echo -e "~giacomo\r\n" >&5
cat <&5

This code opens a TCP socket to tilde.team at port 70 sends the selector string ~giacomo followed by the line terminator (\r\n) and prints the answer. Simple.

You can do some telnet magic instead, which is easier but not as cool as the other³:

telnet tilde.team 70
~giacomo

If you run the code you’ll see you can understand the response with your bare eyes with no parser involved. Isn’t that great?

Notice that in our examples our selector string is ~giacomo. Gopher supports empty strings as selector strings that, in most cases, return a Menu where we can see which selector strings are valid. Why don’t you try it yourself?

Move to Clojure

Now we understand what it’s happening under the hood, it’s time to move to Clojure.

A simple text request can be understood like this piece of Clojure code here (which involves more Java than I’d like to):

; Define the function to make the queries
(defn send-text-request
  [host port body]
  (with-open [sock (java.net.Socket. host port)
              writer (clojure.java.io/writer sock)
              reader (clojure.java.io/reader sock)
              response (java.io.StringWriter.)]
    (.append writer body)
    (.flush writer)
    (clojure.java.io/copy reader response)
    (str response)))

; Make a query and print the result
(println (send-text-request "tilde.team" 70 (str "~giacomo" "\r\n"))

As you see, it’s not waiting to the dot at the end of the file and it’s not doing any kind of parsing, error checking or timeout handling, but it works. This a minimal (and ugly, clean the namespaces!) implementation for you to be able to run it in the REPL.

Binary or not?

The binary is almost the same but the output must be handled in a different way. As Clopher is a terminal based application I made it store the answer in a file.

There’s a simple and beautiful way to handle temporary files in Java that you can access from Clojure. As I wasn’t a Java user before I didn’t know this:

(defn- ->tempfile
  "Creates a temporary file with the provided extension. If extension is
  nil it adds `.tmp`."
  [extension]
  (doto
   (. java.io.File createTempFile "clopher" extension)
   .deleteOnExit))

With this function is really simple to create a temporary file and copy the download there. It’s also easy to ask the user if they want to store the file as a temporary file or in a specific path. With the code below, calling to download-file-to works like we described. If destpath is nil a temporary file is created. Cool.⁴

(defn download-file-to
  [host port srcpath destpath]
  (with-open [sock   (->socket host port)
              writer (io/writer sock)
              reader (io/reader sock)]
    (.append writer (str srcpath defs/CR-LF))
    (.flush writer)
    (io/copy reader
             (io/output-stream
               (or (io/file destpath)
                   (->tempfile (get-extension srcpath)))))))

doto, make Java interop less painful

You probably know what doto does but it’s interesting enough to talk about it here. It returns the result of the first form with all the rest of the forms applied inserting the first form’s result as first argument and discarding the result of the operations. This sounds weird at the beginning but in cases like this one where you are working with mutation it’s really handy:

We are creating a File instance and returning it after calling .deleteOnExit on it. Take in consideration that .deleteOnExit returns nothing, so discarding its return value is great. We want to return the File, not the result of the .deleteOnExit operation.

Once we now how to deal with doto we can improve the caller with this function that creates sockets with some timeout applied that connect automatically:

(defn- ->socket
  ([host port]
   (->socket host port 10000))
  ([host port timeout]
   (doto (java.net.Socket.)
         (.setSoTimeout timeout)
         (.connect (java.net.InetSocketAddress. host port) timeout))))

Replacing java.net.Socket from the example above with a call to this function will make the call handle timeouts, configuring the socket on its creation.

Whatever, right? Better check the code for that. Beware that it may change as I keep going with the development. Maybe not, it depends on the time I spend on this.

Here’s the link to the code. Relevant part can be found in src/clojure/clopher in a file called net or similar:

Link to the repository

It’s time to move on because this is taking longer than it should. We are just warming up, let’s leave it simple at the beginning, there will be chance to make this complex in the near future.

Hope you enjoyed this post.

Hey! But what about the Menus?

Menus are just queried like any other text document so they can be queried with this little code. The parsing, processing and so on is only needed for user interaction so we’ll deal with that later. Don’t worry. We all have to learn to be patient.

See you in the next step.